[Issue #3322] Transform opportunity attachment files #3486
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdragon
reviewed
Jan 15, 2025
babebe
reviewed
Jan 15, 2025
|
Run both locally and tests LGTM!!! Great util functions! |
babebe
previously approved these changes
Jan 15, 2025
…ublished (#3503) ## Summary Fixes #3500 ### Time to review: __10 mins__ ## Changes proposed Handle the following two cases during the transformation process * Opportunity deleted - clean up attachments from s3 * Opportunity stops being a draft - move all the attachments to the other s3 bucket ## Context for reviewers Mostly just some additional file utils for moving files to handle the above scenarios. Only noteworthy callout is that there isn't really a concept of "moving" a file on s3, it's just a copy+delete.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3322
Time to review: 10 mins
Changes proposed
A lot of file utilities (used in this PR) for handling reading/writing/naming files on s3
Utility for setting up s3 file paths for the attachments
Logic to handle inserts/updates/deletes of attachments and the files that need to move around on s3.
Context for reviewers
There are some scenarios I haven't accounted for yet when the opportunity itself is modified (deleted / is no longer a draft), I originally wanted to handle this in a single PR, but I'll split that out as this one already was getting too big.
See the ticket for details on the scenarios we need to handle.
Additional information
Testing this is a bit tedious - there is a lot that needs to be setup exactly to test it.
I'd recommend nuking anything you already have with
make volume-recreateSet the env var to enable the job to run (add
TRANSFORM_ORACLE_DATA_ENABLE_OPPORTUNITY_ATTACHMENT=1to override.env)Run
make consoleand in that dof.StagingTsynopsisAttachmentFactory.create_batch(size=50)and thenexit()Finally you can run the job by doing
make cmd args="data-migration load-transform --no-load --transform --no-set-current"